Ambient triggered notifications

ABSTRACT

A method of rendering a haptic effect on a haptic output device is provided based on an ambient environmental stimulus signal received from a sensor. A notification event based on the stimulus signal is determined. A haptic effect is rendered on a haptic output devices based on the notification event. A system and computer-readable medium are also provided for rendering a haptic effect on a haptic output device based on an ambient environmental stimulus signal.

FIELD

One embodiment is directed to a notification system. More particularly,one embodiment is directed to a notification system that can outputnotifications based on ambient input.

BACKGROUND INFORMATION

Users may receive alerts on their phone or other electronic devices.Typically, users will have a relationship with the entity sending thealert and the alert will typically be generated based on some digitallyreceived information. For example, a user can install a weatherapplication. The weather application can digitally connect to weatherstatus servers, receive an alert, and pass the alert on to the userthrough the application by triggering the phone to beep or vibrate.

Conventional systems provide alerts based on an established relationshipwith the alert giver and use data capabilities of the device to receiveinformation that generates the alert.

SUMMARY

Embodiments include a method of rendering a haptic effect on a hapticoutput device. An ambient environmental stimulus signal can be receivedfrom a sensor. A notification event based on the stimulus signal can bedetermined. A haptic effect can be rendered on the haptic output devicebased on the notification event. Other embodiments include a system andcomputer-readable medium to render a haptic effect on a haptic outputdevice.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a notification system in accordance withone embodiment of the present invention.

FIG. 2 is a block diagram of a notification system in accordance withone embodiment of the present invention.

FIG. 3 is a flow diagram for providing notifications in accordance withone embodiment.

FIG. 4 is a flow diagram for providing a notification with an embeddedsteganographic message in accordance with one embodiment.

FIG. 5 is a flow diagram for receiving and decoding a notificationsignal stimulus containing an embedded steganographic message inaccordance with one embodiment.

FIG. 6 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment.

FIG. 7 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment.

FIG. 8 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment.

FIG. 9 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment.

FIG. 10 is a flow diagram illustrating synchronizing haptic effects inaccordance with some embodiments.

DETAILED DESCRIPTION

Embodiments allow a device user to receive a notification or alertwithout needing a pre-existing relationship with the alert giver andwithout needing a data connection. A stimulus in the ambient environmentcan be sensed by the device. In one embodiment, the device can recognizethe stimulus and provide a notification or alert based on the recognizedstimulus. In one embodiment, the device can decode information embeddedin the stimulus and provide an alert based on the decoded stimulus. Insome embodiments, a data connection can be used to augment the alertinformation. In some embodiments, a relationship between the device userand the alert giver can be previously established that sets preferencesfor the device holder for types of alerts can further augment the alertinformation.

It may be desirable or important to generate an attention-getting alertfor users even in the absence of a pre-existing relationship and even inthe absence of an available data connection on a user's device. Forexample, a store may want to alert shoppers of a sale or specialpromotion. Emergency personnel may want to alert people in a local areaabout the presence of an emergency vehicle or emergency evacuationsituation. A theater may want to alert attendees of the start of a showor the end of an intermission. Typically, to achieve a broadcast alert,a public address system, siren, or other conventional mechanisms (likeflashing house lights) may be used. However, in some circumstances thismay not provide the desired level of attention among users. In somecases, such alerts can lose their effectiveness as an alert as users getused to them and begin ignoring them and treating them as merebackground noise. For example, sirens in New York City can be ubiquitousat times. Also, the alert is limited to the conventional system and doesnot transmit to a device.

In contrast, embodiments described herein encode information insideexisting broadcast media (e.g. audio) so that smart devices cantranslate this information into a notification framework suitable foruse in a one-to-many communications need. In some embodiments, thenotification framework can provide notification events rendered onhaptic output devices, as described below. Notifications can berestricted to an area of broadcast media for a local area (e.g., a mallor city street), but in some embodiments notifications can betransported over existing media streams (e.g., broadcast television) toreach larger audiences.

Several varieties of notifications or alerts can be available dependingon the notification mechanism. Branded alerts can relate to commercialuses for notifying or alerting users within an ambient area to deals andsales for particular stores or brands. Location based reminder alertscan provide notifications or alerts for users within an ambient area ofsome particular event within proximity to the area. For example, usersin a theater can be notified that a show is about to begin simply bybeing in the ambient area of the stimulus that triggers the haptic alertsystem to provide a haptic effect. In another example, a chirpingcrosswalk signal can trigger an alert to a user's device related to thecrosswalk status. In another example, a store in a mall can have themall music system play notifications of a flash sale intended to drivetraffic to the store through an embedded message in the music. In thisexample, the mall music system can be configured to play the alertstimulus embedded in mall music in select speakers in the mall (e.g.,near the store having the sale) so that users in the mall with awearable or mobile devices can get an alert based on their location anddirecting them to the store. Emergency alerts can provide users withinan ambient area of the alert stimulus, or within a broadcast area of thealert stimulus (e.g., within an ambient area of a television or radiobroadcast output), an alert related to an emergency situation, such asan evacuation or threat. An emergency vehicle siren can contain codingthat would provide additional alerts to all users within the ambientrange. This would benefit deaf or audio-distracted users. For example,vehicle sensors can receive siren stimulus and provide a tactile alertto drivers through their seat, steering wheel, or mobile device. Inaddition to these types of alerts, other alerts can include an alert ornotification from an appropriately equipped physical interface to a userthrough the user's wearable, mobile, or personal device withoutrequiring a pre-existing device relationship. For example, a smart lightswitch could be equipped with a speaker to emit an inaudible soundduring user interaction which can be received by the user's wearable andused to stimulate a haptic effect as an alert, perhaps an alert that theswitch will be automatically turning off the lights to warn any user'sin the room to reengage the switch if needed.

A notification or alert can fall into multiple of the above examplecategories, another category, or no category at all. In someembodiments, an alert can be understood as being any information sentusing ambient techniques as described herein to a device that canreceive the information and provide the information using a notificationdevice. An alert can be considered to be an urgent notification.

In some embodiments, the ambient stimulus causing the notification cantrigger the user's device to connect to a cloud service and providepersonal multimodal notifications and/or actions based on the user'sprofile or preferences or relationship with the alert broadcaster. Forexample, after receiving an notification, the user's profile may alterwhat type of special offer the user may receive or may cause an email tobe sent, a social networking site to be updated, a credit card activityto occur, a service to trigger, or some other downstream activity. Insome embodiments, user preferences can be cached on the user's device sothat, if no data connection is available, a personalized notificationcan be provided.

In some embodiments, users can opt-in or opt-out to some kinds ofnotifications, such as branded or location based notifications.Emergency alerts may be prohibited from being opted out. However in someembodiments, even these might be opted out. An opt-in preference canallow a user to receive individualized offers through a data connectionto a cloud or server.

In some embodiments, notifications can be provided to a user's devicevia the device's screen and/or speaker. In some embodiments,notifications can be provided via a haptic output device associated withthe user's device.

FIG. 1 is a block diagram of a notification system 10 in accordance withone embodiment of the present invention. System 10 can include a device30 that includes a haptic output device 18. Internal to system 10 is ahaptic feedback system that generates haptics on system 10. In oneembodiment, a haptic effect is generated by haptic output device 18.System 10 can be a handheld device such as a phone or tablet, a wearabledevice, such as a smart watch or smart glasses, or an accessory devicethat works in conjunction with another device, such as a smart watch orsmart glasses that works in conjunction with other devices. System 10can include a display and/or speakers (not pictured). System 10 can alsobe a device in proximity or in contact with a user, such as a car seat,steering wheel, display, bell, whistle, or any device capable ofproviding a haptic effect that can be perceived.

The haptic feedback system includes a processor or controller 12.Coupled to processor 12 is a memory 20 and a haptic output device drivecircuit 16, which is coupled to a haptic output device 18 located ondevice 30. Haptic output device 18 can include any type of haptic outputdevice, including motors, actuators, electrostatic friction (“ESF”)devices, ultrasonic frequency (“USF”) devices, and any other hapticoutput device that can be used to provide haptic feedback to a user.

Other such haptic output devices 18 may include flexible, semi-rigid, orrigid materials, including smart fluids actuators, rheological fluidicactuators, Macro-Fiber Composite (“MFC”) actuators, Shape Memory Alloy(“SMA”) actuators, piezo actuators, and Micro-Electro-Mechanical System(“MEMS”) actuators.

Processor 12 may be any type of general purpose processor, or could be aprocessor specifically designed to provide haptic effects, such as anapplication-specific integrated circuit (“ASIC”). Processor 12 may bethe same processor that operates the entire system 10, or may be aseparate processor. Processor 12 can decide what haptic effects are tobe played and the order in which the effects are played based on highlevel parameters. A haptic effect may be considered “dynamic” if itincludes some variation in the generation of haptic effects amongsthaptic output device(s) or a variation in the generation of hapticeffects based on a user's interaction with handheld or wearable device30 or some other aspect of system 10, such as user preferences thatspecify parameters for haptic effects based on the stimulus received.

Processor 12 outputs the control signals to haptic output device drivecircuit 16, which includes electronic components and circuitry used tosupply haptic output device 18 with the required electrical current andvoltage to cause the desired haptic effects. System 10 may include morethan one haptic output device 18, and each haptic output device 18 mayinclude a separate drive circuit 16, all coupled to a common processor12. Memory device 20 can be any type of storage device orcomputer-readable medium, such as random access memory (“RAM”) orread-only memory (“ROM”). Memory 20 stores instructions executed byprocessor 12. Among the instructions, memory 20 includes an ambienttriggered haptic effects module 22 which are instructions that, whenexecuted by processor 12, generate drive signals for haptic outputdevice 18 that provide haptic effects based on an ambient stimulus, asdisclosed in more detail below. Memory 20 may also be located internalto processor 12, or any combination of internal and external memory.

System 10 includes one or more sensors 17 that can sense aspects of anambient environment. For example, sensors 17 can includetemperature/humidity/atmospheric pressure sensors to captureenvironmental conditions, an inner field measurement unit (“IMU”) withaccelerometer, gyroscope, and magnetometer to characterize the motion,velocity, acceleration and orientation of the device, a microphone tocapture audio information, an optical sensor to capture visualinformation, and wireless transmitters to receive/transmit informationfrom/to other devices wirelessly. In general, sensor 17 can include anysensing device configured to or adapted to sense an aspect of an ambientinput, such as, but not limited to, an accelerometer, a capacitivesensor, a hall effect sensor, an infrared sensor, an ultrasonic sensor,a pressure sensor, a fiber optic sensor, a flexion sensor (or bendsensor), a force-sensitive resistor, a load cell, a LuSense CPS2 155, aminiature pressure transducer, a piezo sensor, a strain gage, a linearposition touch sensor, a linear potentiometer (or slider), a linearvariable differential transformer, a compass, an inclinometer, amagnetic tag (or radio frequency identification tag), a rotary encoder,a rotary potentiometer, a gyroscope, an on-off switch, microphone,photometer, altimeter, bio monitor, camera, or a light-dependentresistor. In alternate embodiments, system 10 can include one or moreadditional sensors, in addition to sensor 17 (not illustrated in FIG.1). In some of these embodiments, sensor 17 and the one or moreadditional sensors may be part of a sensor array, or some other type ofcollection of sensors. The collection of sensors need not be found onthe same device. For example, sensor input data can includes data from aplurality of sensors associated with the device, such as a sensorassociated with a peripheral device (e.g., smart watch or wearable) ofthe device. Sensor 17 can provide sensor information to processor 12.Sensor information from sensor 17 can be provided to processor 12 byconventional wired or wireless techniques.

In some embodiments, haptic output device 18 is an optional feature ofsystem 10 and notifications associated with the system can be providedvia a display or speaker associated with system 10.

FIG. 2 is a block diagram of a notification system in accordance withone embodiment of the present invention. A environmental stimulus source205 can be output to a transmission medium 210. Devices 220 and 230,such as a smart watch and phone, can correspond to system 10 and canreceive the stimulus through the transmission medium. Optionally,devices 220 and 230 can communicate with each other, such as when onedevice is an accessory of another device. Optionally, devices 220 and230 can communicate to a cloud based or remote server at 240. In someembodiments, information may obtained from a service (e.g., in thecloud) or may operate in a cloud computing environment. A cloudcomputing environment (i.e., the “cloud”) can be an environment in whichcomputing services are not owned but are provided on demand. Forexample, information may reside on multiple devices in a networked cloudand/or data can be stored on multiple devices within the cloud. In someembodiments, a remote server can be used to provide information todevice 220 or 230.

FIG. 3 is a flow diagram for providing notifications in accordance withone embodiment. In one embodiment, the functionality of the flow diagramof FIG. 3 (and FIGS. 4-5 and 9 below) is implemented by software storedin memory or other computer readable or tangible medium, and executed bya processor. In other embodiments, the functionality may be performed byhardware (e.g., through the use of an application specific integratedcircuit (“ASIC”), a programmable gate array (“PGA”), a fieldprogrammable gate array (“FPGA”), etc.), or any combination of hardwareand software.

At 305, a program is provided for use on a device, such as system device10 of FIG. 1. The program may be native to the device, such as anintegral part of the device's operating system programs, a program thatis provided by the original equipment manufacturer, or a programprovided by a third-party creator, such as an application that isuser-installed. Third-party applications may be provided at amarketplace for such applications.

At 310, a stimulus is sensed by the device, such as device 10. Thedevice can be in a continuous sensing state. The stimulus can be anyambient stimulus that a sensor located on the device can sense. In someembodiments, the stimulus can be an audio sound wave sensed by amicrophone. In some embodiments, the stimulus can be an optical inputsensed by an optical sensor (e.g., camera or scanner). In someembodiments, the stimulus can be movement of device 10 sensed by anaccelerometer, gyroscope, or pressure sensor. In some embodiments, thestimulus can be a pressure change related to a pressure profile. Forexample, device 10 could come into direct contact with a surface havinga specific static pressure profile (e.g. an embossed surface) or a usercould construct a gesture that has a specific pressure profile. Thisprofile could be used to identify a notification event that is thenprovided back to the user. In some embodiments, the stimulus can beperceivable by a human, while in other embodiments the stimulus may beunperceivable by a human. For example, for an audio signal, the pitch ofthe audio signal may be outside the sensitivity range of normal humanhearing, which is approximately 16-17,000 Hz, but within the sensitivityof a microphone sensor.

At 315, the stimulus can be processed. In some embodiments, the stimuluscan be processed by fingerprinting the stimulus. For example, an audiosignal can be sampled and fingerprinted. Fingerprinting audio allows theaudio to be recognized when comparing an audio signal's fingerprint to adatabase of audio fingerprints. Any suitable technique can be used forfingerprinting. For example, fingerprinting audio can be done by aprocess similar to the following. First, the audio signal can be downsampled to reduce the number of possible samples. Next, a spectrogram ofthe samples can be calculated. Next, frequency band filtering can beperformed on the spectrogram to further reduce the complexity of thesignal. Filtering can produce logarithmically spaced bins for thespectrogram. Wavelet decomposition can next be used to further reducecomplexity, for example, by applying a standard Haar waveletdecomposition. The larger wavelets can be kept while discarding waveletsthat do not contribute much information. These fingerprints can befurther reduced through minhashing and locality sensitive hashing.

A similar fingerprinting technique can be used for optical or otherambient sensor stimulus. In some embodiments, pattern recognition orother suitable algorithms can be used. For example, a series of bumps ina roadway can produce a pattern that can be sensed by an accelerometeror pressure sensor and processed to determine the user's speed orprocessed to generate an alert about an approaching intersection. Forexample, an embossed surface can have a specific pressure profile whichcan be detected by a pressure sensing device associated with device 10.

In some embodiments, stimulus can be processed by applying an inversesteganographic algorithm to decode a stimulus containing embeddedsteganographic information. Steganographic information encoding canapply to visual and auditory information. A steganographic algorithm canreplace parts of a stimulus with data in a way that is imperceptible toa human. For example, an audio file can have a stegnographic signaturehidden in it by replacing signal information with message information atperiodic intervals. Signal information replaced can include the leastsignificant two bits, for example, so as to minimally impact the audiofile when rendered. The inverse operation can also be applied to extractthe steganographically encoded signature. The audio can be captured andsampled until a steganographic pattern is found. Information can becaptured from the steganographic pattern and used to determine theembedded message. Steganographic data can be made robust to account forlossiness of a captured audio signal. A person of ordinary skill willunderstand that any other available technique can be substituted forsteganography to embed a signature or message in the stimulus audio,video, or other as applicable.

Processing can also include taking the fingerprinted stimulus andcomparing it to a database to classify the stimulus. For example, afingerprinted sound can be used to determine what the sound is. Thesound of a siren can be fingerprinted and compared to a database offingerprints that matches a siren. In the case of a steganographicallyembedded message, processing can also include analyzing the message andcomparing it to a set of known messages or providing the message.

At 320, a notification event is determined based on the processedstimulus. A notification event can include a notification sent to anoutput mechanism of a device, such as device 30. Output mechanisms caninclude a display, speaker, haptic output device, and so forth. Forexample, for a fingerprinted audio stimulus, the recognized audio signalcan correspond to a particular haptic effect for alerting the deviceuser. For example, a recognized siren signal can cause a vibrotactilehaptic effect to be determined and output on a device. In embodimentsusing steganographically embedded messages, the notification event canbe determined based on the embedded message. For example, the messagecan contain a byte-code, text, vibration pattern and so forth. Forexample, the message can contain a code that can be cross-referenced toa haptic effect, audio output, and/or display output signal as anotification event. One or more notification events can be producedbased on the processed stimulus. In some embodiments, users can setpreferences for what type of notification event or haptic effect shouldbe determined based on a fingerprinted audio stimulus orsteganographically embedded message. In some embodiments, a displayassociated with the device or speaker associated with the device canprovide coordinated information. For example, a haptic effect can bedetermined on a phone in response to a siren, the phone can display an“Emergency” notification on the phone's display, and can play an alarmsound on the phone or a connected Bluetooth speaker.

At 325, the notification event or haptic effect is provided on thedevice, such as device 30 of FIG. 1. In embodiments where thenotification event is a haptic effect, the notification event can beprovided by playing or rendering the haptic effect on a haptic outputdevice in the device. In some embodiments, the haptic output device,such as haptic output device 18 of FIG. 1, can be located in a wearabledevice or located nearby.

FIG. 4 is a flow diagram for providing a notification with an embeddedsteganographic message in accordance with one embodiment. At 405, aninput signal is received. At 410, a notification is determined to beincluded in the signal. The notification can be a message or code to alookup table of messages or similar mechanism, such as a database. Insome embodiments, the message or alert can be encrypted for security.

At 415, a steganographic algorithm is applied to embed the notification(i.e., message or code) into the signal to produce a notification signalstimulus including the steganographically embedded message in the inputsignal. Any appropriate steganographic algorithm can be used. Forexample, for an audio signal, a steganographic algorithm using leastsignificant bit coding, parity coding, phase coding, spread spectrum,echo hiding, or other suitable algorithm can be used.

At 420, the notification signal stimulus with the embeddedsteganographic encoding is transmitted. The transmission can be done viaspeaker if the signal is an audio file. The transmission can beaccomplished via a data transmitter (and display if the signal is avisual file). Other embodiments can use other appropriate transmissionmediums.

FIG. 5 is a flow diagram for receiving and decoding a notificationsignal stimulus containing an embedded steganographic message inaccordance with one embodiment. At 505, an input stimulus is receivedthat has an embedded steganographic message (i.e., the notificationinformation as a message or code). At 510, an inverse steganographyalgorithm can be applied. The inverse algorithm can be selected based onthe type of algorithm used to create the steganographic stimulus. Thealgorithm can sample the stimulus to determine how to apply the inversealgorithm. In some embodiments, the decoded message can be decrypted ifit was encrypted.

At 515, a notification event, such as a haptic effect, can be determinedfrom the steganographic message. The notification event can be based onor specified by the notification message or can be looked up in a tableor database that correlates notification messages to notification events(e.g., a code correlated to a haptic effect). User preferences canspecify preferences that can insert or modify lookup entries thatcorrelate notification messages to notification events. At 520, anoptional remote data “cloud” lookup can occur either from a cloud, aserver, or a local cache from a previous cloud or server response. Forexample, a received message can be correlated to preferences located ina server and deliver a notification event based on the remotepreferences. For example, the message can be a code ‘0xFFFF’ thatcorresponds to a coupon offering in a store playing music that has asteganographic message embedded therein as a notification message.‘0xFFFF’ can be looked up locally on a user's device to alert the userto the coupon and provide the message “Sales offering at this store. Seeassociate for details.” With a cloud look up, ‘0xFFFF’ can becross-referenced to additional data stored in the cloud or on a server,such as user preferences, opt-in information, or marketing information,to specify a customized message, such as “Towels are on sale just foryou today Phil—buy 2 or more at 25% off each.” Using a cloud basedlookup, other device information can also be sent to the cloud or serverover the data connection to generate the personalized response. Forexample, the device's International Mobile Equipment Identity (“IMEI”)number (typical in phones and other mobile devices), phone number, GPSlocation, or other available data can be provided to the cloud orserver.

At 525, the notification event can be provided. For example, if thenotification event is an alert, the alert can be sent to the user'sdevice causing a haptic effect, such as a vibration, and a message canoptionally be displayed on the device's display.

FIG. 6 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment. Element 605 is an audio signalrepresented by a waveform diagram. Element 605 is passed to a speaker205 which corresponds to environmental stimulus source 205 of FIG. 2.Broadcast speaker 205 transmits the audio signal as a sound intransmission medium 210, which can be air, water, solids, and so forth.A microphone sensor located in a device corresponding to system 10 ofFIG. 1 receives the audio. The device has a module to fingerprint theaudio at element 620. A notification identifier is determined at module630 based on the fingerprinted audio and notification information, e.g.,information correlating notification event identifiers to notificationmessages. At module 640, a notification event, such as a haptic effect,can be determined based on the notification event identifier. In someembodiments, haptic notification module 640 can use the “InstinctiveAlerts Framework” by Immersion Corp. In some embodiments, a notificationevent can correspond to an audio output or display output sent to aspeaker or display associated with system 10. In some embodiments, anotification event can correspond to a haptic effect that can be playedon haptic output device 18 of FIG. 1 associated with system 10 (such asan actuator on a phone or on an accessory device, such as a wearable orother peripheral). In some embodiments a message related to thenotification event can be displayed on a display of the device (notshown).

FIG. 7 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment. Element 705 is an audio signalrepresented by a waveform diagram. The audio signal of 705 has anembedded steganographic notification message. Element 705 is passed to aspeaker 205 which corresponds to environmental stimulus source 205 ofFIG. 2. Broadcast speaker 205 transmits the audio signal as a sound intransmission medium 210, which is air, water, solids, and so forth. Amicrophone sensor located in a device corresponding to system 10 of FIG.1 receives the audio. The device has a module to apply an inversesteganographic algorithm at element 720 to retrieve an embeddednotification message. The message can be decrypted if it was encryptedfor security. A notification event identifier is determined at module630 based on the embedded message and notification information, e.g.,information correlating notification event identifiers to notificationmessages. At module 640, a notification event can be determined based onthe notification event identifier. In some embodiments, a notificationevent can correspond to an audio output or display output sent to aspeaker or display associated with system 10. In some embodiments, anotification event can correspond to a haptic effect that can be playedon haptic output device 18 of FIG. 1 associated with system 10 (such asan actuator on a phone or on an accessory device, such as a wearable orother peripheral). In some embodiments a message related to thenotification event can be displayed on a display of the device (notshown).

FIG. 8 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment. Element 705 is an audio signalrepresented by a waveform diagram. The audio signal of 705 has anembedded steganographic message. Element 705 is passed to a speaker 205which corresponds to environmental stimulus source 205 of FIG. 2.Broadcast speaker 205 transmits the audio signal as a sound intransmission medium 210, which can be air, water, solids, and so forth.A microphone sensor located in a device corresponding to system 10 ofFIG. 1 receives the audio. The device has a module to apply an inversesteganographic algorithm at element 720 to retrieve an embedded message.The message can be decrypted if it was encrypted for security. Anotification event identifier is determined at module 630 based on theembedded message and notification information, e.g., informationcorrelating notification event identifiers to messages. Module 835 canestablish a connection to a cloud server or other remote server thatuses the message to determine cloud alert information. Cloud basedinformation can include information particular to the user receiving thesteganographic message. At module 840, a notification event, such as ahaptic effect, can be determined based on the cloud alert information.In some embodiments, a notification event can correspond to an audiooutput or display output sent to a speaker or display associated withsystem 10. In some embodiments, a notification event can correspond to ahaptic effect that can be played on haptic output device 18 of FIG. 1associated with system 10 (such as an actuator on a phone or on anaccessory device, such as a wearable or other peripheral). In someembodiments a message related to the notification event can be displayedon a display of the device (not shown).

FIG. 9 is a system diagram illustrating an ambient audio notificationsystem in accordance with one embodiment. In particular, FIG. 9illustrates a system for converting an audio signal into an audio signalwith an embedded steganographic message or signature. Element 605 is anaudio signal. Module 905 is a steganographic algorithm that can beimplemented using a general purpose processor or dedicated processor forembedding a message, signature, or other notification information intothe audio stream. Module 905 can also include an encryption module toencrypt the message or signature prior to embedding. In someembodiments, the signature can be burned into a chip, such as amicrocontroller, FPGA, or the like, that can receive the audio signal,apply the signature, and transmit the audio. Element 705 is an audiosignal with an embedded steganographic message or signature. Forexample, a chirping at a crosswalk can be modified with module 905 witha burned in message or signature to provide a modified chirping soundthat is indistinguishable to hear, but contains a steganographicsignature.

In some embodiments automatic content recognition (“ACR”) can be used tosynchronize haptic effects to a stimulus. ACR techniques are typicallyused to synchronize second screen applications with a media broadcast.Haptic effects or haptic tracks can be synchronized using ACRtechniques. Pattern recognition techniques can be used to detectwatermarks in an audio signal that specify a timing element of the audiosignal. The timing element can be used to estimate the time-stamp of themedia elements of a presentation (i.e., being streamed, played back, ordisplayed as a visual or audio alert element). For example, a song cancontain watermarks that change throughout the song. A watermark W can beexpected at a time t in the song. When watermark W is found, mediaaccompanying the audio can then be synchronized to time t. In addition,a haptic track for the media can be synchronized to the same time-stampestimate. In another embodiment, the coded watermark can contain themedia's timestamp, i.e., the timestamp for the media is coded aswatermark in the media.

In another embodiment, fingerprinting can be used to synchronize hapticeffects to an audio stimulus, such as a broadcast audio sound wave.Fingerprinting can be used as described above to identify a particularstimulus (e.g., an audio track). Fingerprinting can further be used toidentify a timestamp associated with the fingerprinted portion of thestimulus. For example, a portion of an audio stimulus, as received by amicrophone, can be compared to known portions of the stimulus to findwhere in the source audio the fingerprinted portion is. Using thisinformation about the file, a timestamp can be derived from the file. Inanother example consistent with another embodiment, a portion of anaudio stimulus, as received by a microphone, can be compared to a lookuptable (or database or the like) to find the same portion with atimestamp correlated to it.

Embodiments can use ACR techniques to play a haptic track for astimulus. For example, a song can be recognized according to the flow ofFIG. 3, a time-stamp estimated, and a haptic track played (as the hapticeffect alert element) synchronized to the song. For example, the songcould be the National Anthem of the United States, a fight song at asporting event, or some other recognizable song. The haptic track doesnot have to be embedded in the stimulus, but can exist on the device orbe downloaded or streamed from the cloud or a server. In lieu of ahaptic track, distinctive haptic effects (or alerts) can be triggeredbased on the timing estimate of the stimulus (e.g., audio track).

In some embodiments, video can be synchronized to an audio track and ahaptic track. The tracks can be asynchronous and be synchronizedaccording to watermarks or fingerprints. For example, a clip from amovie could be played. An audio stream can be watermarked orfingerprinted to determine an estimated time-stamp for the clip and anassociated haptic track for the entire movie can be skipped to therelevant time according to the estimated time-stamp of the audio clip.

In another embodiment, the device, such as device 10 of FIG. 1, can havean embedded set of haptic effects identified by codes. Instead ofcreating a haptic track or specific haptic effects, information can beembedded using steganographic or other techniques into the audio track.The embedded information can contain haptic effect codes and associatedparameters, such as intensity, duration, relative timing, and so forth.When the haptic playback system detects an embedded effect code andparameters, it can prepare a haptic effect according to the effect codeand parameters and deliver it at a specified relative timing.

FIG. 10 is a flow diagram illustrating synchronizing haptic effects inaccordance with some embodiments. At 1010, an audio input signal isreceived. As disclosed above, the audio input signal may optionally bein a pitch range that is outside the normal range for human hearing. Theaudio signal may have an embedded signature or watermark specifying atime code, haptic effect code, and parameters. In some embodiments, timecode information can be estimated from a fingerprint of the audiosignal. At 1020, a pattern, fingerprint, watermark, or signature can bedetected in the audio signal. At 1030, a time-code can be estimated fromthe pattern, fingerprint, watermark, or signature. In some embodiments,1030 can be skipped, such as when an effect code for a haptic effect isembedded in a watermark or signature. At 1040, if a haptic track isused, the haptic track can be skipped to the estimated time code. Ifhaptic effects are matched to time codes, a haptic effect can beretrieved for the estimated time code. If a haptic effect code is used,then a haptic effect can be looked up according to the haptic effectcode and optional parameters can be set for playing the haptic effect.At 1050, the haptic effect is played on the haptic output device, suchas haptic output device 18 of FIG. 1.

One example of the use of the flow of FIG. 10 is where a user iswatching a movie on television. The user's wrist wearable device can beembedded with some haptic tracks related to different movies. Oncapturing and identifying a specific audio watermark in the movie, theACR system synchronizes the haptic track to the media and starts playingback the related haptic track/effects on the wearable. Another exampleis where a user is watching the Stanley Cup final game on his tabletwhich is augmented with haptic playback features as described herein.Haptic effects parameters related to the game stream are embedded aswatermarks in the audio signal of the game. When analyzing the audiosignal, the system can detect the watermark and playback a hapticeffect, with the specific parameters.

As disclosed, embodiments provide a system for generating a notificationevent, such as a haptic effect alert, that can accompany a visual oraudio alert on a smart phone, smart device, or peripheral. Anenvironmental ambient condition can provide a stimulus that is detectedby a sensor on the device. The stimulus can be processed for patternrecognition, fingerprinting, watermarking, or signature information. Theinformation can be matched to a database of known patterns orfingerprints, or in the case of watermarking or signature embedding,information can be extracted specifying alert or haptic information. Theprocessed information can be passed to the notification system thatcauses a notification event to occur on the device. Some embodiments donot require a previous relationship or data connection. Some embodimentscan use a data connection and/or preferences established from a previousrelationship to augment or customize the alert output. Some embodimentscan synchronize haptic effect information with an audio signal usingthese techniques.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

1. A method of rendering a haptic effect, comprising: receiving anambient environmental stimulus signal from at least one sensor;determining a notification event based on the stimulus signal; andrendering a haptic effect on a haptic output device based on thenotification event.
 2. The method of claim 1, wherein the stimulussignal comprises an audio input, a motion input, or a visual input. 3.The method of claim 2, wherein the audio input is outside a normal rangefor human hearing.
 4. The method of claim 1, wherein the notificationevent is associated with an alert.
 5. The method of claim 1, wherein thedetermining comprises: taking a fingerprint of the stimulus signal;comparing the fingerprint to a database of fingerprints; and selectingthe notification event based on the fingerprint.
 6. The method of claim1, wherein the determining comprises: detecting an embeddedsteganographic message in the stimulus signal; decoding thesteganographic message into a decoded message; and selecting thenotification event based on the decoded message.
 7. The method of claim1, further comprising: estimating a time-code for the stimulus signal;and selecting the haptic effect to render based on the estimatedtime-code.
 8. The method of claim 1, wherein the at least one sensor isassociated with a peripheral device.
 9. A system for rendering a hapticeffect, comprising: a receiver module configured to receive an ambientenvironmental stimulus signal from at least one sensor; a process moduleconfigured to determine a notification event based on the stimulussignal; and a haptic output device configured to render a haptic effectbased on the notification event.
 10. The system of claim 9, wherein thestimulus signal comprises an audio input, a motion input, or a visualinput.
 11. The system of claim 10, wherein the audio input is outside anormal range for human hearing.
 12. The system of claim 9, wherein thenotification event is associated with an alert.
 13. The system of claim9, wherein the process module is further configured to: take afingerprint of the stimulus signal; compare the fingerprint to adatabase of fingerprints; and select the notification event based on thefingerprint.
 14. The system of claim 9, wherein the process module isfurther configured to: detect an embedded steganographic message in thestimulus signal; decode the steganographic message into a decodedmessage; and select the notification event based on the decoded message.15. The system of claim 9, further comprising a time-code estimationmodule configured to estimate a time-code for the stimulus signal,wherein the process module is further configured to determine the hapticeffect based on the estimated time-code.
 16. The system of claim 9,wherein the at least one sensor is associated with a peripheral device.17. A computer readable medium with instructions stored thereon torender a haptic effect, the rendering comprising: receiving an ambientenvironmental stimulus signal from at least one sensor; determining anotification event based on the stimulus signal; and rendering a hapticeffect on a haptic output device based on the notification event. 18.The computer readable medium of claim 17, wherein the stimulus signalcomprises an audio input, a motion input, or a visual input.
 19. Thecomputer readable medium of claim 18, wherein the audio input is outsidea normal range for human hearing.
 20. The computer readable medium ofclaim 17, wherein the haptic effect is associated with an alert.
 21. Thecomputer readable medium of claim 17, wherein the determining comprises:taking a fingerprint of the stimulus signal; comparing the fingerprintto a database of fingerprints; and selecting the notification eventbased on the fingerprint.
 22. The computer readable medium of claim 17,wherein the determining further comprises: detecting an embeddedsteganographic message in the stimulus signal; decoding thesteganographic message into a decoded message; and selecting thenotification event based on the decoded message.
 23. The computerreadable medium of claim 17, further comprising: estimating a time-codefor the stimulus signal; and selecting the haptic effect to render basedon the estimated time-code.
 24. The computer readable medium of claim19, wherein the at least one sensor is associated with a peripheraldevice.